In large-scale distributed storage systems, clients may upload large amounts of data on a regular basis. For example, in a typical cluster or data center, clients may upload several hundred terabytes of data every day. To store clients' new data, distributed storage systems generally add new hardware (e.g., hosts) on a regular schedule, which may mean adding several racks of new hardware at each data center every week. A challenge large-scale distributed storage systems face with this is that a majority of new data tends to be processed and stored to the newly added hardware, because the old hosts are typically nearly full in terms of storage space. From a storage perspective, the new hardware has enough free disk space to accommodate the client data; however, the new hardware may not have enough processing power and resources (e.g., CPU, memory and disk IOPS) to process and store the new data efficiently. At a smaller scale, this may not be a problem as the new hardware may have enough processing power, but at larger scales, when the request load tends to exceed the processing power capacity of the new hardware, this may result in a hot spot (the new hardware) in the distributed storage system, resulting in poor performance (latency and throughput) and higher error rates for client requests. Using higher-end hardware to increase the processing power generates a high business cost and may be wasteful, since once the new hardware is filled up the upload shifts to a next round of newly added hardware.
Another consequence of such a hot spot is that newly uploaded data is generally stored in new hardware added at the same time across two or more data centers, which may create an undesirable strong correlation in data distribution. For example, if there is a defect in a batch of hardware installed across data centers so that the devices in the batch fail in all the data centers, all copies of client data distributed to the different data centers may be lost, which may result in a permanent data loss.
While embodiments are described herein by way of example for several embodiments and illustrative drawings, those skilled in the art will recognize that embodiments are not limited to the embodiments or drawings described. It should be understood, that the drawings and detailed description thereto are not intended to limit embodiments to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope as defined by the appended claims. The headings used herein are for organizational purposes only and are not meant to be used to limit the scope of the description or the claims. As used throughout this application, the word “may” is used in a permissive sense (i.e., meaning having the potential to), rather than the mandatory sense (i.e., meaning must). Similarly, the words “include,” “including,” and “includes” mean including, but not limited to.